Techniques for pointing to locations within a volumetric display

ABSTRACT

The present invention is a system that creates a volumetric display and a user controllable volumetric pointer within the volumetric display. The user can point by aiming a beam which is vector, planar or tangent based, positioning a device in three-dimensions in association with the display, touching a digitizing surface of the display enclosure or otherwise inputting position coordinates. The cursor can take a number of different forms including a ray, a point, a volume and a plane. The ray can include a ring, a bead, a segmented wand, a cone and a cylinder. The user designates an input position and the system maps the input position to a 3D cursor position within the volumetric display. The system also determines whether any object has been designated by the cursor by determining whether the object is within a region of influence of the cursor. The system also performs any function activated in association with the designation.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a divisional of U.S. application Ser. No.10/183,944, filed Jun. 28, 2002, now U.S. Pat. No. 7,324,085, which isincorporated herein by reference. This application is related to andclaims priority to U.S. provisional application entitled User InterfacesFor Volumetric Displays, having serial number 60/350,952, by Kurtenbachet al, filed Jan. 25, 2002, this application is also related to U.S.application entitled Three Dimensional Volumetric Display Input AndOutput Configurations, having Ser. No. 10/183,970, by Kurtenbach et al,filed Jun. 28, 2002, to U.S. application entitled Volume ManagementSystem For Volumetric Displays, having Ser. No. 10/183,966, byKurtenbach et al, filed Jun. 28, 2002, to U.S. application entitledWidgets Displayed And Operable On A Surface Of A Volumetric DisplayEnclosure, having Ser. No. 10/183,945 by Fitzmaurice et al. filed Jun.28, 2002, to U.S. application entitled Graphical User Interface WidgetsViewable And Readable From Multiple Viewpoints In A Volumetric Display,having Ser. No. 10/183,968, by Fitzmaurice et al, filed Jun. 28, 2002,to U.S. application entitled A System For Physical Rotation ofVolumetric Display Enclosures To Facilitate Viewing, having Ser. No.10/183,944, by Balakrishnan et al, filed Jun. 28, 2002, and all of whichare incorporated by reference herein.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention is directed to a system that allows users to pointat objects within a volumetric display system, and, more particularly toa system that allows a number of different pointing approaches andpointing tools.

2. Description of the Related Art

A class of three-dimensional (3D) displays, called volumetric displays,is currently undergoing rapid advancement. The types of displays in thisclass include holographic displays, swept volume displays and staticvolume displays. Volumetric displays allow for three-dimensional (3D)graphical scenes to be displayed within a true 3D volume. That is, avolumetric display is not a projection of volumetric data onto a 2Ddisplay but a true physical 3D volume. Such displays can take manyshapes including cylinders, globes, domes, cubes, etc. with a dome beinga typical shape. Because the technology of these displays is undergoingrapid development those of skill in the art are concentrating on theengineering of the display itself. As a result, the man-machineinterface to or the ways in which people interface with these types ofdisplays is receiving scant attention.

While the volumetric displays allow a user to view different parts of atrue 3D scene, the act of viewing the different parts typically requiresthat the user physically move around (or over) the display or that thedisplay be moved or rotated in front of the user. As the display movesrelative to the user, graphical objects may also move relative to theuser. When the display is relatively stationary or when it is relativelymoving, the user may need to interact with the display by pointing tosomething, such as a model object to, for example, paint the object, orto select the object for some function such as to move the object orselect a control on an interface of the object. The object to which theuser needs to point may be at any level within the display from thesurface of the display adjacent the enclosure to the farthest distancewithin the display from the enclosure or the user. As a result, the userneeds a mechanism for pointing to objects at different locations withina volumetric display. Today, those in the field do not appear to beconcerned with this problem. Because many computer users are familiarwith conventional interface tools and techniques, what is needed is amechanism that will allow users to point at objects within thevolumetric display in a situation where the viewpoint changes and thattakes advantage of the learned behavior of users with respect totwo-dimensional (2D) display interfaces, such as the 2D mouse drivencursor.

SUMMARY OF THE INVENTION

It is an aspect of the present invention to provide a system forpointing at objects within a volumetric display system

It is an additional aspect of the present invention to allow a user topoint at objects from different viewpoints around a volumetric display.

It is another aspect of the present invention to provide a number ofdifferent types of volumetric pointers each having a differentvolumetric geometry.

It is also an aspect of the present invention to provide a number ofdifferent ways in which to point within a volumetric display.

It is an aspect of the present invention to establish a spatialrelationship between the volumetric pointer and the user's bodyposition, specifically the position of their hands. Movements of thehands and body position have a significant spatial congruence with thevolumetric pointer/pointers.

The above aspects can be attained by a system that creates a usermanipulable volumetric pointer within a volumetric display. The user canpoint by aiming a beam, positioning an input device in three-dimensions,touching a surface of the display enclosure, inputting positioncoordinates, manipulating keyboard direction keys, moving a mouse, etc.The cursor can take a number of different forms including a point, angraphic such as an arrow, a volume, ray, bead, ring and plane. The userdesignates an input position and the system maps the input position to a3D position within the volumetric display. The system also determineswhether any object has been designated by the cursor and performs anyfunction activated in association with that designation.

These together with other aspects and advantages which will besubsequently apparent, reside in the details of construction andoperation as more fully hereinafter described and claimed, referencebeing had to the accompanying drawings forming a part hereof, whereinlike numerals refer to like parts-throughout.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a volumetric display.

FIGS. 2A-2B shows tablet input devices associated with the display.

FIG. 2C shows tablets with regions corresponding to the volumetricdisplay.

FIG. 3 illustrates a surface restricted cursor.

FIGS. 4A and 4B show user interaction with the volumetric display.

FIG. 5A and 5B show 3D interaction with the volumetric display.

FIG. 6 show pointing with a beam.

FIGS. 7A-7C show floating cursors.

FIG. 8 depicts hardware of the invention.

FIGS. 9A-9D illustrates several types of digitizer displays.

FIG. 10 depicts a vector based cast ray.

FIGS. 11A-11C show planar based cast rays.

FIG. 12 shows a surface tangent cast ray.

FIGS. 13A and 13B depict a fixed relationship between an input deviceand a ray based cursor.

FIG. 14 shows a cursor of intersecting rays.

FIGS. 15A-15C show a bead cursor.

FIG. 16 depicts a ring cursor.

FIG. 17 illustrates a cone cursor.

FIG. 18 shows a cylinder cursor.

FIGS. 19A and 19B show a plane cursor.

FIGS. 20A and 20B illustrates a region of influence.

FIG. 21A and 21B depict cursor guidelines.

FIG. 22 depicts object control with a ray.

FIG. 23 show user following track pads.

FIG. 24 illustrates the operations for a floating or surface cursor.

FIG. 25 illustrates operations for a ray pointer.

FIG. 26A and 26B illustrate additional pointers

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Volumetric displays allow a user to have a true three-dimensional (3D)view of a scene 12 and are typically provided in the form of a dome 14,as depicted in FIG. 1. The user 16, as can be surmised from FIG. 1, canmove about the dome 14 to view different parts of the scene 12. From aparticular arbitrary viewpoint, a user may want to select an object 18within the scene of the volumetric display and this may be difficult todo with traditional interface tools.

There are a number of different solutions to this problem. Thesesolutions involve creating a volumetric pointer.

A first solution (see FIGS. 2A and 2B) is to restrict movement of acursor type volumetric pointer to a designated plane 30 within thevolumetric display 33 and use a two-dimensional input device 33, such asa stylus pad or mouse to input motion of the cursor on the plane. When astylus and digitizer pad form the input device, the orientation of theplane 30 in the display can be controlled by the pitch of the pad andthe direction of the pad using sensors for sensing pitch and direction.

Another solution (see FIG. 2C) using a 2D digitizer tablet 33 hasdesignated regions on the tablet 33 that map to regions of thevolumetric display 34. For example, a tablet 35 may have a cross-sectionmarked on the tablet such as “Front” 36 and “Back” 37. Placing thestylus in one of these regions maps the cursor to the correspondingposition on the outer shell 37 of the volumetric display 34.Alternatively, having a “Top” 38 and “Front” 39 region delineated on thetablet 40 can position the cursor in 3-space by selecting two points(one in the “Top” region and one in the “Front” region) with the stylus.

Another solution is to restrict the cursor 41 to moving along the outersurface 42 of the display as depicted in FIG. 3. The cursor 41 travelsalong the surface 42 at a point that is the closest point on the surface42 to a stylus 44 even when the stylus 44 is lifted from the surface 42.A surface moving cursor can also be controlled using a touch sensitivedisplay enclosure as well as the arrow keys of a keyboard, a mouse andother 2D input devices. With a surface traveling or restricted cursor, aconvention is used to designate what is selected. The convention limitsthe designation to objects on the surface of the enclosure, to objectsvertically under the point of touch, to a closest object, to objectsorthogonal to the surface at the cursor, etc. Objects within the rangeof influence of the cursor would typically be shown as being within thatinfluence by, for example, being highlighted.

The surface moving cursor can also be used to tumble the contents of thedisplay. For example, as the cursor moves over the top of the display asdepicted in FIG. 3 the contents of the display are locked to the cursorand, thus the contents “tumble” within the display enclosure.

FIGS. 4A and 4B show a user 50 touching the display 52 at two points andthe pointing convention being the creation of vertical virtual planeswhich the user can move by moving the points of touch to, for example,push aside objects that the virtual planes encounter.

A further solution is to allow a user to manipulate a cursor (a flyingor floating volumetric pointer) within the three-dimensional (3D) spaceof the volumetric display using a three-dimensional input device, suchas the tilt mouse set forth in U.S. Pat. No. 6,115,028, the Flock ofBirds system from Ascension Technology Corporation, etc. FIGS. 5A and 5Bsequentially depict a user moving a 3D input device 72 in space adjacentto the display 74 and a cursor 76 in the display moving incorrespondence thereto.

Another solution, as depicted in FIG. 6, is to allow the user to pointat an object 90 to be selected using a three dimensional pointing device92, such as a beam pointer, to thereby point to the object 90 to beselected using a visible volumetric pointer ray 94.

An alternative solution is to partition the volumetric space into a 3Dgrid and use pushbuttons to advance or retard a cursor in each dimension(e.g., using the arrow keys on the keyboard with or without modifierkeys moves the cursor to the next cell in the 3D grid). Additionally,selecting an object can be done by determining a traversal sequencethrough the volume using a heuristic algorithm. For example, consider avolume space that is partitioned into a stack of thin slices or “slabs”.A scan algorithm could search for objects starting at the top left ofthe slab space, scanning across from left to right, row-by-row until thebottom of the slab is reached. This same scan is performed for eachprogressively deep slice of the volumetric space. Again, the net effectof this algorithm is to make each object in the volume addressable bydefining a sequence of objects and having the user jump to the next orprevious object using a “next” key.

These volumetric pointer solutions will be discussed in more detailbelow.

The cursor being moved can perform a number of different functionsincluding designating/selecting objects, changing the display of objectsin the display, such as by applying paint or dye, moving objects withinthe display and other functions typical of cursors used in 2D displays.

The cursor being moved using the approaches noted above can be of anumber of different varieties as discussed below.

The cursor can be a volumetric point cursor, such as small object in thescene like the 3D arrow 110 depicted in FIG. 7A. While this has theadvantage of an easily understood metaphor because 2D arrows are usedwith conventional 2D displays, these cursors can suffer from beingobscured by other objects in the line of sight in the conventionaldisplays. It is often difficult in conventional displays to perceivewhere in the depth dimension the cursor resides. This problem isalleviated in Volumetric Displays due to the enhanced depth perceptionand users wider field of view. Further, since volumetric displays willallow easy scene rotation, this in turn will increase the efficiency ofpointing in 3D with a point volumetric cursor.

The cursor can also be a 3D volume cursor 112 as depicted in FIG. 7B.When used with conventional displays, volumetric volume can enhancedepth perception. The volume cursor shape could be cubic, spherical,cylindrical, cross, arrow or arbitrary shapes like a 3D shovel, tiretube or irregularly shaped object. While depth perception is not aproblem with volumetric displays, volume cursors nonetheless affordcertain advantageous properties when used with volumetric displays.First, if the volume cursor is made semitransparent, objects behind thecursor can still be seen. Second, the volumetric nature of the cursorcan enable volume operations such as selecting multiple objects at once.

The cursor can also be a depth controllable type cursor, such as a beadcursor 114 as depicted in FIG. 7C. A head type depth cursor allows theuser to control the bead of the cursor using two different modes ofinteraction. The cursor is positioned by pointing a beam 116 at anobject and the position of the cursor 114 along the beam is adjustedwith a position control, such as a slider, a thumbwheel, pressuresensor, etc. When the bead 114 contacts or is within an interfacecontrol, such as a button, the control can be activated. The depth typecursor could also have the cursor be a stick or wand shape rather thanthe bead shape shown in FIG. 7C. The stick or wand could be divided intosegments with a different cursor function allocated to each segment andthus be a smart cursor. For example, assume that the cursor has twosegments: a delete segment and a modify segment. During operations, whena “delete” segment contacts an object and the function is activated, thedelete function is performed when the control is activated while whenthe “modify” segment contacts the object the object is modifiedaccording to a predetermined function when the control is activated.

A cursor used for entry of text (2D or 3D) into the volumetric displaywould preferable have an I-beam shape. A convention sets the lay of aline of text

The present invention is typically embodied in a system as depicted inFIG. 8 where physical interface elements 130, such as a rotary domeposition encoder, infrared user position detectors, a keyboard, touchsensitive dome enclosure surface, mouse, beam pointer, beam pointer withthumbwheel, stylus and digitizer pad or stylus and stylus sensitive domeenclosure surface, stylus with pressure sensor, etc. are coupled to acomputer 132, such as a server class machine. The computer 132 uses agraphical creation process, such as the animation package MAYA availablefrom AliaslWavefront, Inc., to create three-dimensional (3D) sceneelements. This process, using position inputs as discussed in moredetail later herein, also creates the virtual interface elements, suchas the 3D point cursor, 3D volume cursor, beam, bead, etc. discussedherein. The display output, including the scene and interface elements,is provided to a conventional volumetric display apparatus 134, such asone that will produce 3D a holographic display.

Pointing to objects within a volumetric display can be effectuated usinga number of different volumetric systems as depicted in FIGS. 9A-9D.These systems operate using the technology included in a conventionalstylus and digitizing tablet or pad input devices. This type oftechnology includes transparent and flexible digitizers capable ofsensing and outputting not only the position of the stylus but also theangle (vector) of the stylus with respect to the digitizer surface andthe distance the stylus is located from the surface. These types ofstyli and digitizers are also capable of inputting a control action,such as is required for activating a control, via switches includedwithin the stylus and sensed by the digitizer/tablet via pressuretransducers and via multiple coils.

As depicted in FIG. 9A, a transparent digitizer 150 (for example, atransparent plastic with an embedded wire sensing grid) is included onan outside surface of a dome shaped enclosure 152. The digitizer 150senses a stylus 154 and provides a position of the stylus to a computer156. The computer 156 produces a volumetric scene, along withdetermining a position of a cursor within the display, and outputs thescene with cursor therein to the display system 158, which produces thescene including the cursor within the display enclosure 152. In FIG. 9B,the digitizer 160 is spaced from the enclosure 162 and can take theshape of a box or a cylinder. In FIG. 9C, the digitizer 164 andenclosure 166 can be box or cylindrically shaped (see also FIGS. 2A and2B). In FIG. 9D, the transparent digitizer 168 is spaced from theenclosure 170 and takes the shape of a familiar rectangular tablet.

As previously discussed, cursor position can be based on athree-dimensional input, such as provided by a digitizing glove, orbased on a pointing device such as a beam pointer. In most applications,the beam can be considered a preview of what will be selected once acontrol, such as a button is used to select the object. Beam pointingcan be divided into a number of categories: vector based, planar based,tangent based beam pointing, object pointing or snap-to-grid.

In vector based pointing, as depicted in FIG. 10, an orientation inputvector 190 for a stylus 192 with respect to the display enclosure 194 isdetermined. This vector 190 is used to cast a ray or beam 196 where theray can be coincident with the vector or at some offset with respect tothe vector. The ray 196 can be invisible or preferably made visiblewithin the volumetric display to aid in the pointing process. The castray or vector is used to determine which voxels within the display tohighlight to make the ray visible. Once the path of the ray 196 is knowna determination can be made as to any objects that the ray encounters orintersects. An object, such as virtual object 198, hit by a ray can, ifdesired, be selected when a control, such as a button on the stylus, isactivated. Note that in certain applications, the ray 196 may changeproperties (such as direction, or shape) when hitting or passing throughan object. For example, a ray passing through a container of water maysimulate the bending effect of a light ray in water.

In planar pointing, a ray is cast orthogonal to a designated referenceplane from a contact point of a stylus with a tablet surface. FIG. 11Aillustrates a ray 220 cast from a stylus contact point 222 to a bottomplane 224 of the display enclosure. FIG. 11B shows a ray 226 cast from acontact point 228 to an arbitrary user defined plane 230. The referenceplane can be specified by the input of planar coordinates by the user orwith a plane designation device (see FIGS. 19A and 19B). In FIG. 1C acast ray 232 can be used to select a first virtual object 234 that theray encounters.

In tangent pointing, a ray 250 (see FIG. 12) is cast orthogonal to aplane 252 that is tangent to a digitizer display enclosure 253 at apoint of contact 254 of a stylus 256 with the digitizer. Once again anyobject encountered by the cast ray 250 can be selected.

In FIG. 12 the point of contact from which the ray is cast or projectedis determined by the position of the stylus. This point from which a rayis cast orthogonal to the surface of the display can be designated usingother devices, such as a mouse or the arrow keys on a keyboard. Forexample, moving a mouse on a mouse pad adjacent to the display 253 canmove a ray projection point cursor in “two dimensions” on the surface ofthe display. That is, the ray projection point cursor is a surfacemoving cursor. Assume for the purpose of this discussion that the mousepad has a front side, a back side, a right side and a left side and thedisplay 253 has corresponding sides. When the mouse is moved from frontto back, the ray projection point cursor is moved along the surface ofthe display 253 from front to back in a proportional movement. This isaccomplished by sampling the 2D inputs from the mouse and moving thecursor along the surface in the same direction and the same distance,unless a scale factor is used to adjust the distance moved on thedisplay surface. In this embodiment, the ray is projected from thecursor into the display orthogonal to the surface at the point of thecursor.

With Object pointing, the ray is cast from the contact point toward theclosest point on a designated object in the scene.

Other operating modes can be engaged, such as “snap-to-grid”, whichconstrain the ray to travel along specified paths in the volume space.

As previously discussed, selection using a beam can be performed in anumber of different ways. The beam can automatically select an object,which it encounters or a particular position along the beam can beselected using a cursor, such as a bead or stick as previouslymentioned. As also previously mentioned, the position of the bead can becontrolled by a position device, such as a thumb wheel or a secondarydevice. It is also possible to fix the bead a predetermined distancealong the beam and allow the position of the stylus to indicate theposition of the bead as shown in FIGS. 13A and 13B. FIG. 13A shows thestylus 270 in contact with the enclosure 272 and the bead 274 positionedwithin the display along the ray 276. FIG. 13B shows the stylus 270 at adistance from the enclosure 272 and the bead 274 in the display at asame constant distance from the stylus 270 along the ray 276.

Another bead based selection mechanism is shown in FIG. 14. In thisapproach, a bead 290 is created at an intersection of beam 292 andsecondary beam 294 cast by separate styli 296 and 298. The secondarybeam 294 specifies the position along the primary beam 292 where thecursor is created based on a region of influence or angular tracking andintersection designation.

When a bead is used as a volume cursor, such as the type that can selectobjects when the object is in the volume of the cursor, the presentinvention allows the size of the bead to be changed as depicted in FIGS.15A and 15B. Initially (see FIG. 15A), before or after a bead 310 is ina desired position along a cast ray 312, a user changes or initiates achange in the size of the bead 310 using an input device, such as athumbwheel on a stylus 314. The size can be continuously varied until itis of a size desired by the user as depicted in FIG. 15B. When the beadcursor has reached the desired size it can be positioned surrounding orcontacting an object or objects 316 that the user desires to select (andexcluding undesired objects 318), as depicted in FIG. 15C. The enlargedbead cursor can be shown with the original size bead 310 as an opaqueobject therein to allow the user to see a position of a center of thecursor and, with a surrounding semitransparent volume cursor 320enclosing the embedded objects 316 which have been selected.

A cursor associated with a ray can take other volume geometric shapes inaddition to the bead or stick shapes previously mentioned. As depictedin FIG. 16, the cursor can take the shape of a ring 340 allowing thecursor to select a swept volume 342 when the stylus is moved from aninitial position 344 to a final position 346. The ring 340 (and volume342) can be made semitransparent or opaque as needed for a particularoperation. Objects inside the volume can be selected for a functionaloperation or the swept volume could itself be acted on when a functionis initiated.

The cursor can also take the shape of a cast cone 360 as depicted inFIG. 16 where the cone can be semitransparent and objects within orcontacting the cone can be selected. The cone can have its apex 362 atthe surface of the enclosure as shown or at some user desired positionalong the orientation vector of the input device as specified by aninput device, such as a stylus thumbwheel.

As another alternative, the volume cursor associated with a cast ray cantake the shape of a semitransparent voxel cylinder or shaft 380 centeredon the cast ray and optionally terminated by the bead 384 as depicted inFIG. 18. FIG. 18 also depicts a situation where the objects within theshaft 380 are rendered transparent so the user can see inside or throughobjects 386 and 387 within the display. Essentially a window into anobject is created. The transparent hose created by the shaft 380 stopsat the head 384. The position of the bead 384 along the ray 382 isadjustable and the bottom of the shaft can have a planar or some othershape.

The cursor used for selecting or designating within a volumetric displaycan also take other shapes, such as the shape of a displaying spanningplane as depicted in FIGS. 19A and 19B. Such an input plane 400 can bespecified by a rule or convention and an input device 402 that can be“parked” at a location on the enclosure and that includes a mechanismfor specifying location and orientation, such as the mechanism foundwithin styluses that can be used to designate a contact point and avector. The rule could, for example, specify that the plane must beorthogonal to a bottom 404 of the enclosure 406, pass through the pointof contact and be parallel with the vector. The plane in addition toacting as a cursor can be used in combination with a ray to form acursor where the cursor would be formed at an intersection of the planeand the ray.

The selection mechanism with respect to cast rays can also include aregion of influence that automatically selects objects within aspecified and variable distance of the cast ray as shown in FIGS. 20Aand 20B. In FIG. 20A four objects 420-426 are within the selectionregion 427 of the ray 428 while one object 430 is not. In this figure a“spread” function is also used which is a spatial nearest neighborheuristic. Based on the currently selected object has it's nearestneighbor determined, etc. FIG. 20B shows the same objects but with onlyobject 420 being within the region of influence.

For point or volume cursors that move within the volume of thevolumetric display 440 it may be helpful to provide visual aids thathelp to show where in the 3D volume the cursor 442 is located usingaxial based guide lines 444 as shown in FIGS. 21A and 21B. Theguidelines 442 are preferably semitransparent voxels that allow objectsbehind the guidelines in the line of sight of the user to be seen. Theguidelines are particularly useful when the cursor 442 is obscured by anobject.

The cursor and its location designation apparatus, such as a stylus canbe used to control objects. For example, an object selected by aray/bead can be moved by activating a move function and moving thelocation of the bead with the stylus. Another possible motion is arotational motion where an object 460 rotates as a stylus 462 selectingthe object rotates as depicted in FIG. 22. Note that that object canrotate along any arbitrary axis. However, most applications willpreferably rotate the object along the axis defined by the input ray.

For situations where the cursor is relegated to movement along thesurface of a display in an enclosure, it is possible to position virtualcontroller/tools on the surface of the display with which the cursorinteracts. FIG. 23 depicts virtual track pads 480 and 482 on the displaysurface that can be used with a surface cursor or a ray. The track padcould also be used to set positions along a ray. Using a motion trackingsystem, the track pads can move with a user as the user moves around thedisplay.

The different types of pointing discussed above require similar butsomewhat different operations as discussed in more detail below.

The pointing operations (see FIG. 24) involve obtaining 500 input valuesfrom the input device where the input values are the raw output valuesof the input device (for example, stylus/pad or glove).

The system then combines 502 the input values with enclosure shapeand/or position. This allows the system to take into account the shapeof the enclosure to use in deriving a positional coordinate. Forexample, when a 2D input tablet is essentially stretched over a domeshaped enclosure, the tablet only reports a 2D position. However, thisposition value is combined with the knowledge of the shape of the dometo derive or map to a 3D position (i.e., a point in three space which ison the dome). This shape and/or position information allows the correctmapping between input and output spaces to occur. Note that not all ofthe different embodiments make use of the shape of the enclosure. Forexample, when the input device senses its 3D location, the shape of theenclosure does not need to be known. However, the position of thedisplay relative to the sensing volume of the 3D input device needs tobe known. Hence, this operation also factors in display and enclosureposition.

Once the positional coordinate is known a cursor position metaphor forthe input and output spaces is applied 504. This is used because thecursor control techniques can be much more than simply a 3D position inspace but may involve metaphors such as “ray-casting” that useadditional information. For example, if a stylus based ray with a depthcontrollable bead is used, the ray is projected from the positionalcoordinate of the contact point of the stylus with the enclosure alongthe orientation vector of the stylus. The depth of the bead set by thedepth control device (slider, thumbwheel, etc.) is used to determine apoint along the ray from the contact point at which to create the beadcursor. As another example, for a surface cursor, the applied metaphorinvolves transforming or mapping the input device coordinates (such asthe coordinates of the stylus above or on the surface of the enclosure)into volumetric display surface coordinates and finding the closestpoint (voxel) on the display surface to the input coordinates as theposition. For a floating cursor, the input device coordinates (such asthe 3D position of a glove in a space adjacent the display enclosure)are mapped to a corresponding 3D position within the display. A floatingcursor is typically used with a dome shaped display surrounded by avolume sensing field. For the sake of this floating cursor discussion,the display has a back and a front. A cursor is at some position in thedisplay. The input device, such as a non-vector flock-of-birds sensor,has a button that activates the “move” function of the device. If thecursor is at some arbitrary position in the display and the user isstanding in front of the display, the input device is activated and ismoved toward the display, the cursor moves from front to back. If theuser turns off the activation, moves to the rear of the display,activates the device and moves the device toward the display, the cursorwill move from the back to the front of the display. That is, movementof the input device away from the user will always move the cursor awayfrom the user. The metaphor in this situation requires that themovements of the cursor be matched in orientation and distance to themovement of the glove, unless a scale factor is involved where, forexample, movement distance of the cursor is scaled to ½ of the movementof the glove. The metaphor may also involve separate input surfacesbeing used depending on the shape of the display. For example, acylinder can be comprised of 2 separate input surfaces: one for the topof the cylinder, and one for the side. In general, a mapping functionbetween the input coordinate space for the volumetric display and theaddressable voxel space within the volumetric display can be defined fora desired metaphor.

The cursor is then positioned 506 (see FIG. 24) at the appropriateposition within the volumetric display.

Once the cursor is positioned within the display, a determination ismade 508 as to whether the cursor (volume or point or influence orshaped) is “contacting” any object, such as a model object or a virtualinterface. This involves mapping from the cursor position within thedisplay to an appropriate virtual space. A point cursor is just a pointin 3 space and has no orientation or volume information. If the point iscoincident with an object contact exists. Things like volume cursors,influence cursors, ring cursors, etc., can require orientationinformation as well as volume information. The points comprising volumeobjects in the display space need to be compared to the pointscomprising the oriented volume cursor to determine if contact exists.

Once the mapping has occurred, a determination 510 is made as to whetherthis is a smart cursor or a control having a particular function thathas been activated. If so, the system determines 512 whether thefunction has been activated, such as by the depression of a button on astylus. If the function has been activated, it is performed 514 and thesystem continues inputting input device coordinates.

For the ray based pointing, once the input device coordinates are input630, such as the position of a stylus on the enclosure surface, adetermination 632 is made concerning an origin of the ray that is to becast, as depicted in FIG. 25. In a stylus contact type mode (see FIGS.10, 11A and 13B) a transformation similar to that performed for thesurface restricted cursor is performed. For a vector mode, a closestpoint on the surface of the display along the vector is determined. Thesystem then casts 634 a ray. For the vector mode, the ray corresponds tothe vector. For the planar mode, a search is performed for a point on areference plane at which an orthogonal will intersect the ray point oforigin. If the ray has a displaced origin, such as for a displaced conethe origin of the ray is displaced accordingly. Next, the systemdetermines whether an object has been contacted 636. When the ray is theselection mechanism (see FIG. 11C), conventional ray casting objectdetection along the ray is performed and the first object encountered,if any, is flagged. When a bead is used, the bead is treated like avolume cursor as discussed above. Once a contact determination is made,the system performs the operations 506-510 discussed previously.

When a plane, such as depicted in FIG. 19, is used as a cursor itdefines a virtual plane. This virtual plane can have an orientation andposition in object space. When the virtual plane is activated, objectsthat intersect or come in contact with the virtual plane are alsoactivated. When the virtual plane moves position or orientation, theactivated objects move a corresponding distance and directionproportional to the motion of the virtual plane. Releasing the virtualplane also deactivates the objects which are currently in contact withthe virtual plane. If the resulting virtual plane motion causesactivated objects to be move beyond the imaging chamber of the display,their data structures are still affected even though they are notvisible in the display. Alternative strategies for plane operationinclude: (1) manipulation of volumes of space not just objects intowhich the plane bumps, (2) do not move volumes but instead compressspace, (3) different behavior when the virtual plane cuts across theentire volume or, alternatively, partially intersects the volume, (4) ifthe virtual plane is manipulating volumes of space, differentbehaviors/actions happen depending on whether the space is in front ofor behind the virtual plane. For example, space in front of the virtualplane may compress. However, space behind the virtual plane can eitherbe enlarged or empty space can be defined.

The system also includes permanent or removable storage, such asmagnetic and optical discs, RAM, ROM, etc. on which the process and datastructures of the present invention can be stored and distributed. Theprocesses can also be distributed via, for example, downloading over anetwork such as the Internet.

The present invention has been described using typical devices, such asa stylus to designate objects. However, it is possible to substituteother types of devices that can include pointing mechanisms, such asspecialized surgical tools, which would allow the simulation andpractice of a surgical operator.

The rays of the present invention have been shown as typically beam orpencil type rays. The rays can also take other shapes, even fancifulones like the cork screw 520 of FIG. 26A and the lightening bolt 522 ofFIG. 26B.

Combinations of the embodiments are also possible. For example, asurface restricted cursor could produce a target ray.

The many features and advantages of the invention are apparent from thedetailed specification and, thus, it is intended by the appended claimsto cover all such features and advantages of the invention that fallwithin the true spirit and scope of the invention. Further, sincenumerous modifications and changes will readily occur to those skilledin the art, it is not desired to limit the invention to the exactconstruction and operation illustrated and described, and accordinglyall suitable modifications and equivalents may be resorted to, fallingwithin the scope of the invention.

1. A volumetric display method, comprising: producing athree-dimensional scene in a volumetric display; and producing athree-dimensional pointer for an object in the display by mapping froman input device position into the display independent of scene contentand display mechanism along a ray cast responsive to the input device,and wherein the pointer comprises a point cursor.
 2. A method as recitedin claim 1, wherein the point cursor is always produced in the display.3. A method as recited in claim 1, wherein the point cursor issemitransparent.
 4. A volumetric display method, comprising: producing athree-dimensional scene in a volumetric display; and producing athree-dimensional pointer for an object in the display by mapping froman input device position into the display independent of scene contentand display mechanism, wherein the pointer comprises a point cursor; andproducing orthogonal guidelines intersecting the point cursor in thedisplay.
 5. A volumetric display method, comprising: producing athree-dimensional scene in a volumetric display; and producing athree-dimensional pointer for an object in the display by mapping froman input device position into the display independent of scene contentand display mechanism along a ray cast responsive to the input device,and wherein the pointer comprises a volume cursor.
 6. A method asrecited in claim 5, wherein the volume cursor has a semitransparentsurface.
 7. A computer readable storage controlling a computer byproducing a three-dimensional scene in a volumetric display andproducing a three-dimensional pointer for an object in the display bymapping from an input device position into the display independent ofscene content and display mechanism along a ray cast responsive to aninput device.
 8. A display apparatus, comprising: a volumetric displaysystem; an input device inputting position coordinates; and a computerproducing a three-dimensional scene displayed by the display system andproducing a three-dimensional volumetric pointer positioned in the sceneresponsive to the position coordinates by mapping from an input deviceposition into the display independent of scene content and displaymechanism along a ray cast responsive to the input device.
 9. A displayapparatus as recited in claim 8, wherein the input device comprises onof a stylus/digitizing surface adjacent a display enclosure, astylus/digitalizing surface on the display enclosure and athree-dimensional input device, and the three-dimensional pointercomprises one of a display surface restricted cursor, a floating pointcursor, a floating volume cursor, a ray cursor, a ray cursor having aregion of influence and a plane cursor.
 10. A display, comprising: avolumetric display scene; and a volumetric pointer within the sceneproduced by mapping from an input device position into the displayindependent of scene content and display mechanism along a ray castresponsive to the input device.
 11. A method, comprising: simulating aline segment in three dimensional space; moving the simulated linesegment in three dimensional space by manipulating an input device; anddisplaying in a volumetric display an intersection of the simulated linesegment with the volumetric display.